es_out: support Upstream Servers with configuration overriding #7608

mabrarov · 2023-06-25T08:42:03Z

Implementation of Upstream feature for the Elasticsearch output plugin.

This pull request is based on pull request #1560 and Forward output plugin.

It was tested in a local setup with:

Single Elasticsearch instance.

Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.

fluent-bit.conf Fluent Bit configuration file used for the test:

[SERVICE]
    Flush           10

[INPUT]
    Name            dummy
    Dummy           { "message" : "this is dummy data" }
    Rate            10

[OUTPUT]
    Workers         4
    Name            es
    Host            localhost
    Port            9201
    TLS             Off
    Retry_Limit     6
    Replace_Dots    On
    HTTP_User       elastic
    HTTP_Passwd     elastic
    Index           fluent-bit

Debug log is available at flb_es.log.

Elasticsearch cluster of 3 master-eligible/data and 1 coordinating nodes.

Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.

fluent-bit.conf Fluent Bit configuration file used for the test:

[SERVICE]
    Flush           10

[INPUT]
    Name            dummy
    Dummy           { "message" : "this is dummy data" }
    Rate            10

[OUTPUT]
    Workers         4
    Name            es
    Upstream        ./upstream.conf
    TLS             Off
    Retry_Limit     6
    Replace_Dots    On
    HTTP_User       elastic
    HTTP_Passwd     elastic
    Index           fluent-bit

upstream.conf Fluent Bit configuration file used for the test:

[UPSTREAM]
    name        elasticsearch-balancing

[NODE]
    name        elasticsearch1
    host        localhost
    port        9201
    Index       fluent-bit-000002
    buffer_size 4M

[NODE]
    name        elasticsearch2
    host        localhost
    port        9202
    Index       fluent-bit-000003
    compress    gzip

[NODE]
    name        elasticsearch3
    host        localhost
    port        9203
    Index       fluent-bit-000004

Debug log is available at flb_es_upstream.log.

Testing

Example configuration files for the change can be found in mabrarov/elastic-stack repository under fluent-bit-es-cluster directory.
Debug log output from testing the change - see above.
Attached Valgrind output that shows no leaks or memory corruption was found - refer to flb_run_code_analysis.log for the output of command
```
TEST_PRESET=valgrind ./run_code_analysis.sh
```
[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature - refer to es_out: support Upstream Servers with configuration overriding fluent-bit-docs#1143.

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

mabrarov · 2023-06-26T07:47:46Z

plugins/out_es/es.h


-struct flb_elasticsearch {
+struct flb_elasticsearch_config {


The whole idea / implementation and naming of new types follows Upstream support in Forward output plugin.

mabrarov · 2023-06-26T07:51:56Z

plugins/out_es/es_conf_parse.c

+        return 0;
+    }
+
+    toks = flb_utils_split((const char *)cloud_auth, ':', -1);


Not:

Suggested change

toks = flb_utils_split((const char *)cloud_auth, ':', -1);

toks = flb_utils_split((const char *) cloud_auth, ':', -1);

because https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md requires to follow https://httpd.apache.org/dev/styleguide.html which requires to not put a space in case of type cast:

The Guidelines
...
There is no whitespace between a cast and the item modified (e.g., "(int)j" and not "(int) j").

mabrarov · 2023-06-28T09:19:30Z

Hi reviewers,

Is it possible to approve only workflow for this pull request, so that automated checks and build can start?

Thank you.

PettitWesley · 2023-06-28T17:55:39Z

@mabrarov sure

mabrarov · 2023-06-29T04:35:15Z

Hi @PettitWesley,

It looks like all failed checks are around run-macos-unit-tests jobs and caused by the following failed unit tests:

flb-rt-in_event_test
flb-rt-out_tcp

I feel like other pull requests have the same issues, i.e. it doesn't seem that the failed checks are caused by this pull request changes.

Help of maintainers is appreciated.

Thank you.

mabrarov · 2023-07-07T11:20:02Z

plugins/out_es/es_conf.c

+    }
+    tmp = flb_upstream_node_get_property(FLB_ES_CONFIG_PROPERTY_LOGSTASH_PREFIX, node);
+    if (tmp) {
+        ec->logstash_prefix = (char *)tmp;


Type cast is illegal. Should use flb_sds_create instead. Working on fix in feature/out_es_upstream_support_extended_test branch.

mabrarov · 2023-07-10T10:10:13Z

Hi @PettitWesley,

Is it possible to trigger automated workflow (build) for this pull request one more time? I found & fixed one issue and added tests for the new code since last build happened.

Thank you.

mabrarov · 2023-07-20T15:16:09Z

Hi dear reviewers,

Is it possible to get this pull request reviewed / accepted sooner? Is there something pending / waiting from my side to start review?

Thank you.

mabrarov · 2023-08-22T07:18:01Z

Hi @PettitWesley and @edsiper,

It feels like you are code owners for Elasticsearch output plugin. Is there something pending / waiting from my side to start review of this pull request? This new feature was requested 4 years ago and I feel it is something which multiple users of Fluent Bit (not just my team) would like to have.

Thank you.

mabrarov · 2023-10-26T18:01:48Z

Hi @PettitWesley and @edsiper,

Is there a chance to get this pull request reviewed (and eventually accepted)? Please let me know if something is missing and blocking that.

Thank you.

116davinder · 2023-12-10T17:34:15Z

Please someone review this, I also want this feature to merged and released.
thank you!

dioniseo · 2024-02-28T16:16:57Z

+1

Hello reviewers, could you please review this PR?

This feature also will be very useful in our project and very useful feature to support HA.

Thank you!

mabrarov · 2024-03-22T20:21:46Z

Just FYI, my team successfully tested replacement of following log shipping solution:

Fluent Bit DaemonSet in K8s (Forward output plugin with Upstream Servers, Upstream node per Fluentd instance) → Fluentd (2 instances with fluent-plugin-elasticsearch configured to connect to all nodes of Elasticsearch cluster) → Elasticsearch cluster (3+ nodes)

with:

Fluent Bit DaemonSet in K8s (Docker image built from source branch of this pull request, Elasticsearch output plugin with Upstream Servers, Upstream node per Elasticsearch node) → Elasticsearch cluster (3+ nodes)

For my team it means (this pull request means) a possibility to get rid of 7-9 Fluentd instances (where each Fluentd instance occupies a single VM).

github-actions · 2024-06-27T01:52:22Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

mabrarov · 2024-06-27T06:13:40Z

This pull request is still actual and up-to-date (ready for merging, tested with changes coming from target branch).

AzureLeMoon · 2024-07-22T10:27:11Z

Hello, @PettitWesley and @edsiper.

i hope you guys can take a look at this pr and merge it sooner, it's a highly requested feature with its mentioned in the official docs as well.

mabrarov · 2024-07-28T11:38:13Z

Hi @PettitWesley and @edsiper,

It looks like there is one more enhancement which this pull request brings comparing to approach "Fluent Bit (Forward output plugin with Upstream Servers) → multiple Fluentd instances → multiple Elasticsearch instances of a single Elasticsearch cluster":

Fluentd can generate unique ID of Elasticsearch document before inserting data into Elasticsearch. This approach can help to avoid duplicating records (at Elasticsearch side) when re-sending data to Elasticsearch at Fluentd side. Refer to https://github.com/uken/fluent-plugin-elasticsearch?tab=readme-ov-file#generate-hash-id.
Unfortunately, elasticsearch_genid Fluentd filter plugin won't help to avoid duplicated records when Fluent Bit re-sends data to Fluentd.
With this pull request we can generate unique ID of Elasticsearch document at Fluent Bit side - refer to Generate_ID key of Elasticsearch Fluent Bit output plugin configuration. This approach (Fluent Bit built from this pull request, Fluent Bit configured with Elasticsearch output plugin with Upstream Servers and Generate_ID configuration key set to On → multiple Elasticsearch instances of a single Elasticsearch cluster) fully eliminates possibility to get duplicated records at Elasticsearch side due to re-sending of data.

fluent/fluent-bit#7608

mabrarov · 2024-08-22T20:08:03Z

Hi @PettitWesley,

Could you (or someone from maintainers / reviewers) please approve workflow for this pull request, so that the automated checks and build of this pull request can start?

Thank you.

fluent/fluent-bit#7608

…wn to parser of Upstream node configuration section are implemented, e.g. "host" and "port" Signed-off-by: Marat Abrarov <[email protected]>

…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <[email protected]>

…llback contract Signed-off-by: Marat Abrarov <[email protected]>

…with Upstream node configuration For Elastic cloud authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: cloud_id. For AWS authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: http_proxy, no_proxy, tls*. Signed-off-by: Marat Abrarov <[email protected]>

Signed-off-by: Marat Abrarov <[email protected]>

…ended

mabrarov requested review from PettitWesley, edsiper, niedbalski, leonardo-albertovich, fujimotos and koleini as code owners June 25, 2023 08:42

github-actions bot added the docs-required label Jun 25, 2023

mabrarov mentioned this pull request Jun 25, 2023

es_out: support Upstream Servers with configuration overriding fluent/fluent-bit-docs#1143

Open

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 812020d to ba3382a Compare June 25, 2023 09:06

mabrarov mentioned this pull request Jun 25, 2023

es_out: support Upstream Servers #1560

Open

mabrarov commented Jun 26, 2023

View reviewed changes

mabrarov temporarily deployed to pr June 28, 2023 17:55 — with GitHub Actions Inactive

mabrarov temporarily deployed to pr June 28, 2023 18:22 — with GitHub Actions Inactive

mabrarov commented Jul 7, 2023

View reviewed changes

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from ba3382a to b7cd81b Compare July 8, 2023 10:22

mabrarov temporarily deployed to pr July 10, 2023 12:05 — with GitHub Actions Inactive

mabrarov temporarily deployed to pr July 10, 2023 12:28 — with GitHub Actions Inactive

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from b7cd81b to b81d3f7 Compare July 20, 2023 19:38

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from b81d3f7 to f6431c2 Compare September 30, 2023 13:32

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from f6431c2 to d03cde0 Compare December 24, 2023 17:12

github-actions bot added the Stale label Jun 27, 2024

github-actions bot removed the Stale label Jun 29, 2024

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from d03cde0 to 30a2822 Compare August 19, 2024 18:52

mabrarov added a commit to mabrarov/elastic-stack that referenced this pull request Aug 19, 2024

Aligned Fluent Bit configuration with the one which used for testing of

d5d70fd

fluent/fluent-bit#7608

mabrarov added a commit to mabrarov/elastic-stack that referenced this pull request Aug 29, 2024

Aligned Fluent Bit configuration with the one which used for testing of

ade06c9

fluent/fluent-bit#7608

mabrarov force-pushed the feature/out_es_upstream_support_extended branch 3 times, most recently from 7aaea9f to b13a6f8 Compare September 5, 2024 15:25

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from b13a6f8 to f7d0a15 Compare September 6, 2024 09:39

mabrarov added 5 commits October 25, 2024 03:23

pipeline: outputs: es: basic support of Upstream, only parameters kno…

02854d7

…wn to parser of Upstream node configuration section are implemented, e.g. "host" and "port" Signed-off-by: Marat Abrarov <[email protected]>

engine: support of tests which require generation of context passed t…

138e9be

…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <[email protected]>

pipeline: outputs: es: fixed tests after modification of formatter ca…

b97b88e

…llback contract Signed-off-by: Marat Abrarov <[email protected]>

pipeline: outputs: es: tests for upstream node configuration properties

f023976

Signed-off-by: Marat Abrarov <[email protected]>

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from f7d0a15 to f023976 Compare October 25, 2024 04:04

Merge branch 'fluent:master' into feature/out_es_upstream_support_ext…

ce00434

…ended

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

es_out: support Upstream Servers with configuration overriding #7608

es_out: support Upstream Servers with configuration overriding #7608

mabrarov commented Jun 25, 2023 •

edited

Loading

mabrarov Jun 26, 2023

mabrarov Jun 26, 2023 •

edited

Loading

mabrarov commented Jun 28, 2023

PettitWesley commented Jun 28, 2023

mabrarov commented Jun 29, 2023 •

edited

Loading

mabrarov Jul 7, 2023 •

edited

Loading

mabrarov Jul 8, 2023

mabrarov commented Jul 10, 2023

mabrarov commented Jul 20, 2023

mabrarov commented Aug 22, 2023

mabrarov commented Oct 26, 2023

116davinder commented Dec 10, 2023

dioniseo commented Feb 28, 2024

mabrarov commented Mar 22, 2024

github-actions bot commented Jun 27, 2024

mabrarov commented Jun 27, 2024

AzureLeMoon commented Jul 22, 2024

mabrarov commented Jul 28, 2024 •

edited

Loading

mabrarov commented Aug 22, 2024

	toks = flb_utils_split((const char *)cloud_auth, ':', -1);
	toks = flb_utils_split((const char *) cloud_auth, ':', -1);


		struct flb_elasticsearch {
		struct flb_elasticsearch_config {

es_out: support Upstream Servers with configuration overriding #7608

Are you sure you want to change the base?

es_out: support Upstream Servers with configuration overriding #7608

Conversation

mabrarov commented Jun 25, 2023 • edited Loading

mabrarov Jun 26, 2023

Choose a reason for hiding this comment

mabrarov Jun 26, 2023 • edited Loading

Choose a reason for hiding this comment

mabrarov commented Jun 28, 2023

PettitWesley commented Jun 28, 2023

mabrarov commented Jun 29, 2023 • edited Loading

mabrarov Jul 7, 2023 • edited Loading

Choose a reason for hiding this comment

mabrarov Jul 8, 2023

Choose a reason for hiding this comment

mabrarov commented Jul 10, 2023

mabrarov commented Jul 20, 2023

mabrarov commented Aug 22, 2023

mabrarov commented Oct 26, 2023

116davinder commented Dec 10, 2023

dioniseo commented Feb 28, 2024

mabrarov commented Mar 22, 2024

github-actions bot commented Jun 27, 2024

mabrarov commented Jun 27, 2024

AzureLeMoon commented Jul 22, 2024

mabrarov commented Jul 28, 2024 • edited Loading

mabrarov commented Aug 22, 2024

mabrarov commented Jun 25, 2023 •

edited

Loading

mabrarov Jun 26, 2023 •

edited

Loading

mabrarov commented Jun 29, 2023 •

edited

Loading

mabrarov Jul 7, 2023 •

edited

Loading

mabrarov commented Jul 28, 2024 •

edited

Loading